Insight_Extractor

BERT + regex insight extractor with dynamic keyword stemmer.

insight-extractor is a Python 3.12+ library that combines transformer-based semantic search with high-performance regex pattern matching to extract structured insights from unstructured text. Designed for threat-intelligence, OSINT, and security-focused NLP pipelines.

Features

Dynamic Keyword Stemmer — Configurable stemming (Porter, lemmatization, prefix, suffix, fuzzy, or raw regex) with automatic pattern generation for large keyword lists.
BERT Semantic Scoring — Sentence-level relevance scoring using sentence-transformers (all-MiniLM-L6-v2 by default).
Regex Pattern Extraction — Pre-built patterns for CVE IDs, SHA256/MD5 hashes, IP addresses, crypto wallets, onion domains, email addresses, Telegram handles, ransom amounts, file extensions, data sizes, ports, years, and percentages.
Dynamic Keyword Expansion — TF-IDF + cosine similarity automatically grows the keyword bank from input text.
State Persistence — Keyword bank, frequencies, and categories saved to JSON between runs.
Lazy Model Loading — BERT model only loads when semantic extraction is triggered; regex/keyword pipeline runs without it.
Pydantic v2 Models — Type-safe, validated output schemas throughout.

Requirements

Python 3.12 or newer
CPU-only inference supported (no GPU required)

Installation

Step 1 — Clone or unzip the project

cd C:\Users\YourName\Downloads
:: unzip Insight_Extractor.zip here, then:
cd Insight_Extractor

Step 2 — (Recommended) Create a virtual environment

python -m venv .venv
.venv\Scripts\activate

Step 3 — Install with pinned dependencies (most reliable)

pip install -r requirements.txt -c constraints.txt
pip install -e .

This installs the known-good pinned versions from constraints.txt, avoiding the transformers compatibility issue described below.

Alternative (Lighter version in this repo) — install dev dependencies too

pip install -e ".[dev]"

Known Issue — `ModelLoadError: name 'init_empty_weights' is not defined`

Cause: sentence-transformers model loading needs init_empty_weights from the accelerate package. If accelerate isn't installed at all, this error appears.

Fix — run this in cmd then retry:

pip install "accelerate>=1.3.0"

This project's requirements.txt and constraints.txt already include accelerate to prevent this on fresh installs.

Known Issue — `ImportError: cannot import name 'TorchTensorParallelPlugin' from 'accelerate.utils'`

Cause: an accelerate version older than 1.3.0 installed alongside transformers>=4.53.0. Transformers' trainer module imports TorchTensorParallelPlugin, a symbol accelerate only added in 1.3.0. This fires at import time — as soon as anything imports sentence_transformers — not just on model load.

Fix — run this in cmd then retry:

pip install "accelerate>=1.3.0" --upgrade

requirements.txt and constraints.txt pin accelerate>=1.3.0 (currently 1.14.0 in constraints.txt) to prevent this on fresh installs.

Running the Extractor

Basic usage — pass a text file

python -m insight_extractor my_report.txt

Run with no file (uses built-in demo text)

python -m insight_extractor

The demo text contains ransomware, OSINT, CVE, and AI-pipeline content — useful for verifying the install works end-to-end.

Double-click launcher (Windows)

Create run.bat in the project folder:

@echo off
cd /d "%~dp0"
python -m insight_extractor test.txt
pause

Or a drag-and-drop version — drag any .txt onto this .bat:

@echo off
python -m insight_extractor %1
pause

Output

Every run produces two output files in the current directory (or output_dir if set via API):

File	Description
`insights_extracted.md`	Full Markdown report — all entity types, semantic hits, key sentences, keyword stats
`insight_extractor_state.json`	Persisted keyword bank, frequencies, categories — reloaded on next run

Console output sections printed on every run:

=== REGEX ENTITIES ===
=== DYNAMIC KEYWORD MATCHES ===
=== SEMANTIC KEYWORD HITS (top 10) ===
=== KEY SENTENCES ===
=== DYNAMIC EXPANSION: +N new keywords ===
Total tracked keywords: N
Results saved to: insights_extracted.md
=== KEYWORD STATS ===

Regex Patterns — What Gets Extracted

These run on every input with no BERT model required:

Pattern Label	Matches	Example
`CVE_ID`	CVE identifiers	`CVE-2026-48710`
`IP_ADDRESS`	IPv4 addresses	`192.168.1.254`
`HASH_SHA256`	64-char hex strings	`3b4c5d6e...`
`HASH_MD5`	32-char hex strings	`d41d8cd9...`
`DOMAIN`	Domains (.com/.net/.onion/etc.)	`ransom.onion`
`EMAIL`	Email addresses	`threat@dark.io`
`BTC_WALLET`	Bitcoin wallet addresses	`1A1zP1eP5Q...`
`RANSOM_AMOUNT`	Dollar amounts with scale	`$5 million`
`FILE_EXTENSION`	Malware-relevant extensions	`.exe`, `.locked`, `.ps1`
`DARK_WEB`	`.onion` domains	`abc123.onion`
`TELEGRAM_HANDLE`	@handles (5+ chars)	`@threatactor`
`PORT_NUMBER`	Port references	`port 4444`
`TB_GB_DATA`	Data volume mentions	`8 TB`, `500 GB`
`YEAR`	4-digit years 20xx	`2026`
`PERCENTAGE`	Percentage values	`94.3%`

Python API — Full Options